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ABSTRACT 


A highly structured evaluation plan is essential to implementing a successful science, technology, engineering, and math (STEM) intervention program designed to 
improve diversity in STEM education and careers. Effective program evaluations facilitate program improvement and sustainability goals. This article reviews 
paramount program components and discusses the importance of integrating program components in designing a dynamic evaluation plan. Evaluation plan 
development must involve dialogue with program team members and participants before, during, and after the STEM intervention. Translation of raw data to 
actionable intelligence using appropriate analytical techniques is indispensable to maximize the insights derived from an effective evaluation plan. 
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INTRODUCTION 

This article offers a straightforward approach to program evaluation for STEM 
initiatives designed to broaden participation in the STEM pathway. Program 
evaluation is essential to academic and professional development programmatic 
interventions. The entire project team should play a role in the program 
evaluation's development, implementation, and completion. The program 
evaluation may be complicated and involve many elements; however, all 
program evaluations encompass at least four main components. The four 
components include: 1) the determination of the scope of the STEM program, 2) 
the formation of the goals and objectives of the program, 3) the development of 
critical metrics and initiation of measurement strategies, and 4) the review and 
response to evaluation data. 


Business, education, and organizational management journals have highlighted 
the importance of benchmarks and indicators. Benchmarks and indicators are 
essential elements of effective evaluation algorithms. The practical difference 
between a benchmark and an indicator is that benchmarks are goals to 
accomplish or standards that serve as a basis for comparing with other programs, 
usually based on best practices and empirical precedent. Indicators represent a 
more specific way of articulating objectives or goals, typically involving 
numerical and time components. Develop benchmarks by studying the metrics of 
similar successful STEM programs. Develop process-oriented, strategic, and 
organizational performance benchmarks. Consult other articles to learn the most 
advantageous way to visualize benchmarks and indicators. 


Establish benchmarks and critical performance indicators (CPIs) for individual 
program team members, program participants, and program components. 
Moreover, benchmarks and indicators allow quick performance quantification to 
determine adherence to goals, objectives, outcomes, and impacts. Benchmark 
and indicator formulation provides the fundamental framework to elucidate 
essential program information to enhance the STEM program. Perform a needs 
analysis to determine the baseline for benchmarks and indicator establishment. 
Consider benchmarks as long-term goals and indicators as short-term goals used 
to monitor the progress of benchmark completion. Use benchmarks to evaluate 
semesterly or yearly success and indicators to track weekly or monthly 
adherence to program objectives. Failure to achieve goals and objectives 
highlights key areas of concern and potential improvement. 


Logic Model 

A logic model is a valuable visual representation that illustrates the 
interconnections between components of a program (e.g., resources, 
intervention, evaluation, and post-evaluation) (Jones et al., 2020). Logic models 
are used in academic and business environments (Cooper et al., 2020; Strycker, 
2016) and constructed using different designs. Typically, a logic model depicts 
the problem, inputs, outputs (e.g., strategies, activities, participants), outcomes, 
assumptions, barriers, impacts, and evaluation plan. The logic model is a 
blueprint that documents how program resources and actions are connected to 
achieve the desired outcomes and institutional or societal impacts. The logic 
model schematic can also help communicate program goals to various 
stakeholders. As previously mentioned, typically, the logic model includes the 
program evaluation plan; however, a logic model can also be designed 
exclusively for the program evaluation. Figure | shows a generalized logic 
model adaptable to any STEM program framework. Table 1 details specific 
inquiries that must be addressed and included in the logic model. 


Program Title: 


Problem: 


Inputs: 


Outputs: 


Outcomes: 


Impacts: 


Assumptions/Barriers: 


Evaluation: 


Table 1. Logic Model Elements, Definitions, and Questions 


Logic Model Elements 


Definitions 


Questions 


Problem Primary issues, needs, or |What is the nature of the problem or 
priorities addressed by issue that requires a positive 
the STEM initiative. solution? 

Inputs Human, financial, What resources do I have? What 
infrastructural, resources do I need for a successful 
intellectual, and material |STEM program? 
resources. 

Outputs Strategies, activities, What specific methods or activities 
participants, research will be employed during the 
methods, or stakeholders. }intervention? Who are my primary 

participants, audience, or 
beneficiaries? 

Outcomes Effects of a successful What are the short-term, 
implementation. intermediate, 

and long-term goals of the 
program? 

Impacts Future positive impacts | Did the intervention result in 


Assumptions/Barriers 


Evaluation 


with broad implications. 


Perceptions, beliefs, or 
ideas about the STEM 
broadening participation 
initiative. Implicit or 
explicit impediments to 
program completion. 


Determination of STEM 
program effectiveness. 
Strategies to monitor 
adherence to the mission, 
vision, goals, and 
objectives. 


significant societal changes? Did 
the program have consequential 
impacts on education, the economy, 
policy, or the workforce? 


What are the potential assumptions 
or obstacles that could negatively 
affect successful implementation? 


Have the goals and objectives been 
achieved? How can the STEM 
program be improved now and in 
the future? 


Figure 1. Generalized adaptable STEM program logic model outline. 


STEM PROGRAM 


Before developing a specific evaluation plan, consider paramount institutional 
and program elements. Those elements include formulating the vision statement, 
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mission statement, goals and benchmarks, objectives and indicators, and 
program activities. Once these items are clearly defined, establishing the 
evaluation plan becomes a much simpler process. 


Vision 

The STEM program vision statement offers a prosperous global overview of the 
program's future aspirations. Your program's vision statement must be consistent 
and aligned with the institutional vision statement. The program vision statement 
is usually brief and comprises one or two sentences. The vision statement will 
allow for continual refocusing of team members during the program. Sample 
vision statements could include: “To create career opportunities for all students,” 
“To stimulate learning and research to transform society,” “To inspire the next 
generation of scientific talent,” and “To provide access to information to change 
the world.” 


Mission 

In contrast to the vision statement, the program mission statement is typically 
longer, such as five-ten sentences, and provides more details about how your 
program or organization will actualize its vision. The mission statement is a 
broad and concise statement that provides an overview of the broadening 
participation program that documents the strategic goals and purpose. It gives 
more specific information regarding what the STEM program must do to pursue 
its aims. Both the program team and stakeholders benefit from a well-crafted 
mission statement. The internet is replete with excellent academic institutions 
and company mission statements. To better understand the best components for a 
program mission statement, read several mission statements online and apply 
similar ideas to the program's mission statement. It is also important to ensure 
that each member of the project team and institutional administrators play a role 
in developing the mission statement to enhance buy-in. 


Goals 

Setting program goals is not easy and should once again require tremendous 
input from the entire team and external stakeholders. Goals represent 
programmatic endpoints or mark a desired result. In the system discussed herein, 
goals or outcomes may be short-term, intermediate, and long-term. Goals 
represent a distillation of the mission statement. The goals or outcomes section is 
different from the vision statement and mission statement because this section 
allows project officials to present a realistic view of likely programmatic 
outcomes following effective implementation. The vision and mission 
statements are less specific than goals or outcomes statements which are more 
specific. In addition, to being outcome-oriented, they can also be process- 
oriented. The key is ensuring that the tenets in this section reflect the true nature 
of the program components and activities and that the goals and outcomes are 
achievable. 


Objectives 

Objective statements are more specific than goal statements and present 
measurable and actionable program directives to achieve goals. Each goal 
statement should include at least three-five objective statements. Create SMART 
statements. SMART objectives refer to constructing ideas that are specific, 
measurable, achievable, relevant, and completed within a certain timeframe. 
These statements contain categorical and continuous variables that are easy to 
measure using standard or advanced statistical procedures. Use precise 
indicators to monitor adherence to program objectives. 


Table 2. Common STEM Program Indicators 
Project Areas |Indicator Target Action 
Recruitment 1 Project website traffic 1. Increase the number of new 
2. Online applications visitors by 10% monthly. 
3. Recruitment emails 2. Increase the number of 
4. Recruitment presentations applicants by 20% monthly. 
3. Increase the number of 
emails sent by 10% weekly. 
4. Increase the number of 
program recruitment 
presentations by 10% 
monthly. 
Program 1. Comprehension of 1. Improve post-test scores on 
concepts and skills quantitative surveys. 
2. Supply order fulfillment |2. Decrease the time of receipt 
3. — Infrastructure of materials by 10% yearly. 
enhancements 3. Increase the number of 
4. Partnerships and instruments by 25% yearly. 
collaborations 4. Increase the number of new 
letters of collaboration by 
10% quarterly. 
Evaluation 1. Formative and summative |1. Increase the number of 
reports formative evaluation reports 
2. Internal evaluation by 10% monthly and 
3. External evaluation decrease the completion time 
4. Benchmarks and indicators of the summative (e.g., 
annual) report by 20% yearly. 
2. Increase internal evaluator 
meetings by 10% yearly. 
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3. Increase external evaluator 
meetings by 10% yearly. 

4. Develop new benchmarks 
and indicators and modify 
existing statements. 


The design of participant learning objectives and program learning objectives is 
paramount to effective evaluation plans for STEM initiatives. Objective 
statements should reflect the program requirements and expectations and be 
expressed concisely and unambiguously. Well-written objectives also provide a 
harbinger of potential assessment methods to measure compliance. 
Communicate program objectives to program participants on the first day of the 
program and include them in online materials. Construct objective statements to 
focus on what participants will learn, do, or understand from the broadening 
participation program. Clear, unambiguous, accurate directives also provide 
recommendations for direct program implementation strategies and activities. 
Most course syllabi contain information that specifies what students should 
know or be able to do after the course has ended. Consultation of online course 
syllabi in STEM disciplines will present some excellent ideas for quality 
objective statements. 


Moreover, use objective statements to create indicators during the program to 
monitor goal attainment. Effective indicators involve numerical measurements 
tracked before, during, and after the program, such as participant recruitment, 
enrollment, internships, research experiences, number of graduates, target 
GPAs, seminars, workshops, research equipment, publications, number of new 
job hires, and number of new program, departmental, and institutional 
collaborations. Objectives and indicators must be quantifiable and measurable 
and include target actions (Table 2). These and other critical performance 
indicators help keep the team focused on the metrics that matter. 


Strategies 

After you have crafted the project vision statement, mission statement, goals, and 
objectives, it is time to develop and document the nuts and bolts of the program. 
Establish strategies that result in satisfying aims within the target time frame. 
Program activities should be extremely detailed. Research the literature for best 
practices to facilitate successful interventions. Each objective statement should 
include three-five specific strategies to accomplish the objective. Specify which 
team member will be responsible for a particular task. Make sure they know how 
to report metrics and document completion correctly. 


Program Evaluation 

After the components of the STEM initiative, such as vision, mission, goals, 
objectives, and strategies, are elucidated, documented, and implemented, it is 
time to utilize quantitative and qualitative methods to assess adherence to your 
program's vision, mission, goals, and objectives. Generally, program evaluations 
involve observational research and empirical assessments to produce evidence 
of STEM program compliance and effectiveness. The evaluation plan may 
involve the periodic measurement of academic success, achievement, 
perceptions, motivation, career development skills, research skills, pedagogical 
skills, communication skills, and other critical outcomes that determine 
educational and career success in STEM (Flowers, Flowers, & Moore, 2016). 
Incorporate a social scientist into the program team to facilitate the program 
evaluation. Social scientists understand basic and advanced evaluation methods 
and should lead the discussion on best practices in assessment, analytical 
procedures, and data presentation methods. 


The program evaluation must involve multiple formative and summative phases. 
Formative evaluation components refer to adherence or program monitoring 
procedures performed during the program. For example, if your STEM program 
runs for the entire academic year (e.g., Fall and Spring semesters), your 
formative evaluation may occur weekly, monthly, or semesterly. In an 
educational context, formative assessments typically include quizzes, 
homework, or daily journals. Depending on the nature of the STEM program, 
formative assessments could assess program participants, team members, or 
other aspects of the initiative. Formative assessments facilitate reactive feedback 
from participants and staff. Formative evaluations allow program directors and 
principal investigators to make substantive, immediate, and continuous 
instructional or programmatic changes in real time. 


Summative evaluations, on the other hand, occur at the end of the course, year, or 
program. Summative monitoring procedures check for adherence to outcomes 
and impacts. For example, suppose a STEM program runs for the entire 
academic year for three years. In that case, the summative evaluation may occur 
yearly or several years after the program ends. Formative assessments provide 
formal or informal estimates of program participant progress. Summative 
assessments provide formal (e.g., post-test) estimates of global understanding 
and impacts after completing the STEM experience. Formative assessments 
allow teams to determine what participants learn or perceive during STEM- 
based interventions. Summative assessments will enable teams to determine the 
extent of the learning gains. Summative assessments also produce data that 
encourages accountability and generates strategies for future implementation. 
Authentic assessments such as skills-based tasks, scientific communication 
tasks, and comprehensive questionnaires are the preferred participant 
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summative assessments to confirm proficiency and competency in key program 
areas (Sabtiawan, Yuanita, & Rahayu, 2019). For both formative and summative 
assessments, rubrics and clearly defined benchmarks may be helpful to measure 
knowledge, perceptions, and skills adequately (Chan & Ho, 2019). 


As mentioned in the introduction section, a comprehensive STEM program 
evaluation plan must include four common elements: 1) the determination of the 
scope of the STEM program, 2) the formation of the goals and objectives of the 
program, 3) the development of critical metrics and initiation of measurement 
strategies, and 4) the review and response to evaluation data. After the data is 
collected, the important task is to sift through the copious amounts of data to 
determine if the measurement strategies were effective and to meticulously 
probe the data to assess program efficacy. It is essential to thoroughly examine 
the research literature to uncover critical questions, knowledge gaps, mixed 
methods instruments, and issues that disciplinary experts deem important for 
participant development and success (Okulu & Oguz-Unver, 2021). Conducting 
or locating meta-analyses and systematic reviews may help to plan a practical 
evaluation framework. Understanding how the literature's critical issues and 
unresolved questions align with the objectives and outcomes of the program may 
lead to helpful evaluation questions and a robust evaluation plan. 


Integrate online organizational work management application platforms such as 
Monday.com, Asana.com, Trello.com, and Proofhub.com to improve evaluation 
plan efficiency and keep team members on the same page. Features on these 
platforms allow for daily electronic communications and visually appealing 
task-based solutions to enhance accountability. While the platforms mentioned 
above are subscription based, which may not be cost-effective, using Google 
Calendar and other free scheduling applications can also be useful. Further, using 
electronic data collection platforms such as Survey Monkey and Google Forms 
is the preferred strategy for collecting assessment data. Paper-based data 
collection systems lead to reductions in accuracy, response rates, productivity, 
and actionable intelligence. The increased accessibility of mobile phones and 
other mobile technology allows project directors to reach potential survey 
respondents quickly. 

Consider oversampling and power analysis procedures when collecting critical 
data that may determine the effectiveness of the STEM initiative (Rickles, 
Zeiser, & West, 2018). Student attrition may negatively impact evaluation 
metrics. Following the successful implementation of the STEM program 
evaluation, document the successful execution of the evaluation plan in the form 
of educational literature. Furthermore, following a review of the formative and 
summative assessment data, utilize the data to fix problems or create new 
solutions to ensure adherence to program goals and objectives. Table 3 provides 
examples of evaluation schemes. 


Table 3. Evaluation Literature 


Initiative Target Implementation References 
Audience 
Curriculum Pre-service Teachers STEM _ |Dede (2022); Tiirk, Kalayci, 


initiative. & Yamak (2021)a positive 
solution? 
Academic Success Undergraduates Howley, Campbell, Cowley, 
& Cook (2022); Windsor & 
Ivey (2018) 
Research Experiences |Undergraduates Pitre, MIsna, & Mlsna (2022); 


Zhao (2022) 


Brace et al. (2018); Reeves, 
Claydon, & Davenport (2022) 


Graduate Programs Graduate Students 


Bradford, Beier, & Oswald 
(2021); Greer, Chi, & Hylton- 
Patterson (2023)policy, or the 
workforce? 


Bridge Programs Undergraduates 


Mentoring STEM Faculty and 


Undergraduates 


Bradley & Mead (2022); 
Rockinson-Szapkiw, Sharpe, 
& Wendt (2022) 


CONCLUSION 

Broadening participation in STEM education and careers to create a more 
equitable and prosperous society remains a paramount and elusive vision. Data 
published by the National Science Foundation demonstrates that despite 
enormous gains in enhancing diversity at every step of the STEM pathway, 
additional work is required to achieve goals that impact underrepresented 
populations. Develop and implement beneficial STEM initiatives such as career 
development workshops, undergraduate research experiences, coding boot 
camps, summer academic bridge programs, and undergraduate, graduate, and 
faculty mentoring programs that produce impactful metrics. 


Initiatives designed to improve diversity and training opportunities for 
underserved communities are meaningful and have enhanced numerous student 
achievement measures and professional outcomes. Undergraduate research 
experiences continue to be a frontrunner of beneficial strategies to help students 
actualize their true potential and guide career decisions (Borrego et al., 2021; 
Bruthers & Matyas, 2020; Ghebreyessus et al., 2022; Yang et al., 2019). More 
effective program evaluation schemes could elucidate better approaches to 
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integrate undergraduate research into the STEM curriculum. Moreover, 
additional research is needed to study diverse evaluation plans to develop best 
practices for programs involving minority students and minority-serving 
institutions. 


STEM program evaluation involves the development of program aspirations and 
carefully constructing specific strategies for accomplishing program goals. 
Specific techniques include using qualitative questionnaires and quantitative 
surveys designed to collect detailed information from target audiences in a 
specified period. It also involves tracking, monitoring, and adhering to 
compliance and correcting activities to fix the program if things are not trending 
in the right direction. Evaluation systems must apply evaluative strategies to 
assess outcomes and implementation procedures (Onyura et al., 2022). Be 
conscious of reciprocity and interconnectivity when considering the evaluation 
plan. Understand that every component of the STEM program ultimately 
impacts the bottom line. Thus, the comprehensive evaluation scheme must 
contain objectives to evaluate every participant, strategy, stakeholder, 
organizational factor, and objective for programmatic effectiveness. The 
operational definitions of benchmarking and indicators herein provide a basic 
overview of these terms regarding program evaluation; however, hiring a 
consulting firm specializing in creating and measuring benchmarks and 
indicators may be necessary and pragmatic. Remember to budget for the 
program evaluation activities. Since program evaluation is paramount to success 
and sustainability, budgeting around 1%-5% or more of the direct costs to 
complete program evaluation tasks each funding year may be prudent. 


In summary, effective STEM program evaluations are valuable mechanisms to 
demonstrate the programmatic activities that are working and the methods that 
are not working. Incorporating formative and summative assessments will create 
opportunities for making changes during the program that impact success and 
guarantee compliance with project goals, outcomes, and impacts. Creating 
several rubrics to assist the evaluation process is a sound practice to enhance 
evaluation endeavors. Quality rubrics are helpful evaluation tools to facilitate 
formative and summative assessment of goal attainment, provide helpful 
feedback, and ensure comprehension of program expectations. Rubrics guide the 
nature of the evaluation process and should be constructed and disseminated to 
project team members and participants prior to the start of the program. 
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